!build

!test --pybench

Can you check how does the benchmark look like? Make sure there are no unexpected results.

!test --pybench-full --dev

see results here when pipeline finishes

!test --pybench-full --dev

!test --pybench-full --dev

!test --pybench-full --dev

We still see some performance differences between Thunder and nvfuser manual definitions, for some operators, which may be due to slight differences in the fusion definitions generated. I will look into this operator-wise.

I have compared the measured timings against nsys timings for some operators to verify the accuracy of the benchmark infra. My recommendation is to check in these changes adding Thunder benchmarks, while we investigate the performance gap.

My only concern is, would the disruption rendering our benchmark not reliable until we fix all these issues? and does it cause issues on folks looking at the benchmarks?

The thunder.jit performance numbers match against nsys. They also compare correctly for operators like scale-bias-relu, softmax, silu-mul against the manual nvfuser definitions. But for others like normalization, the performance is different. So my next step is to actually compare the fusion definitions, and account how each difference in the 2 fusion definitions correspond to the performance gap (for instance, the manual fusion definitions have some missing downcasts for intermediate ops, or the ordering of ops etc).

So by performance gap, I imply the latter case. The measurements of the benchmarks should be accurate.
This PR is actually fixing one of the measurement issues existent in the backward benchmarks due to grad accumulation.

I think there're still quite some questions on what's the next step and how to automate things like IOBytes computation and stuff.

Agreed! Let me open an issue tracking the IOByte case. I will create an issue for the differences between thunder.jit and nvfuser manual definition performance shortly (I want to do some initial analysis).

!build

PR Reviewer Guide 🔍

(Review updated until commit 083d89d)

Here are some key observations to aid the review process:

def unary_bwd_torch(inputs: List):  # [output, grad_out, fwd_inputs]
    inputs[0].backward(inputs[1], retain_graph=True)

DEFAULT_EXECUTORS = ["eager", "torchcompile", "thunder"]

def setup():
    clear_l2_cache()
    if device == "cuda":
        for inp in inputs:
            if isinstance(inp, torch.Tensor):
                inp.grad = None
        return [inputs], {}

!test --pybench-full --dev

@jacobhinkle @liqiangxl pinging for review.

We continue to see performance difference between Thunder-nvfuser and nvfuser. The thunder-nvfuser executor will not run by default so we can continue investigating this in parallel. This PR has the fix for bwd grad accumulation issue, so we can merge this. If the consensus is to hold off on adding Thunder-nvfuser benchmarks altogether, I can remove it from the executors list for benchmarks as well. That change is minimal.

Latest benchmark run: http://nv/euP

1. nvFuser performance remains the same.
2. Eager and torch.compile remain the same for fwd and are faster for bwd.
3. Thunder-nvfuser and nvfuser performance is different in some cases, mostly bwd pass. See Issue Performance gap between manual nvfuser definition and thunder.jit #3629 for an example.

@jjsjann123 Can you give me the numbers you expect for RoPE bwd after the grad accumulation issue is resolved -- I can verify the numbers with this PR?

!build